Skip to content

Introduce batch.experimental() while server-side batching is in beta #1765

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 81 commits into
base: main
Choose a base branch
from
Open

Introduce batch.experimental() while server-side batching is in beta #1765

wants to merge 81 commits into from

Conversation

@tsmith023
Copy link
Collaborator

@tsmith023 tsmith023 commented Jul 31, 2025 •
edited
Loading

orca-security-eu[bot]
orca-security-eu bot reviewed Jul 31, 2025
View reviewed changes
Copy link

@orca-security-eu orca-security-eu bot left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Infrastructure as Code high 0   medium 0   low 0   info 0 View in Orca
Passed Passed SAST high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Vulnerabilities high 0   medium 0   low 0   info 0 View in Orca

@codecov-commenter
Copy link

codecov-commenter commented Jul 31, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 30.52209% with 346 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.14%. Comparing base (7acf5c0) to head (fa2ef0d).
⚠️ Report is 8 commits behind head on main.

Files with missing lines Patch % Lines
weaviate/collections/batch/base.py 15.28% 266 Missing ⚠️
weaviate/collections/classes/batch.py 32.14% 19 Missing ⚠️
integration/test_batch_v4.py 50.00% 11 Missing ⚠️
weaviate/connect/v4.py 21.42% 11 Missing ⚠️
integration/test_rbac.py 42.85% 8 Missing ⚠️
weaviate/collections/batch/collection.py 65.21% 8 Missing ⚠️
weaviate/collections/batch/batch_wrapper.py 77.41% 7 Missing ⚠️
weaviate/collections/batch/client.py 63.15% 7 Missing ⚠️
weaviate/collections/batch/grpc_batch.py 75.00% 4 Missing ⚠️
weaviate/exceptions.py 55.55% 4 Missing ⚠️
... and 1 more
Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #1765      +/-   ##
==========================================
+ Coverage   86.91%   87.14%   +0.22%     
==========================================
  Files         273      273              
  Lines       19089    19531     +442     
==========================================
+ Hits        16591    17020     +429     
- Misses       2498     2511      +13

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@weaviate-git-bot
Copy link

Great to see you again! Thanks for the contribution.

beep boop - the Weaviate bot 👋🤖

PS:
Are you already a member of the Weaviate Slack channel?

tsmith023 added 17 commits August 5, 2025 14:07
@tsmith023
Update images in CI
7d13908
@tsmith023
Update 1.33 image in CI
ef790f0
@tsmith023
Alter test or lazy shard loading for new potential server behaviour
3b4d99d
@tsmith023
Change other lazy loading test too
6dfd1d7
@tsmith023
Fix CI image for 1.33
689665b
@tsmith023
Update protos, fix setting of batch client in wrapper to avoid races …
c474309 
…with connection
@tsmith023
Remove debug assert in test
72353ef
@tsmith023
Update new batch to use different modes with server, update CI image
63bbdd1
@tsmith023
Refactor to changed server batching options
dd575ed
@tsmith023
Throw error if using automatic batching with incompatible server
9f10a70
@tsmith023
Add exponential backoff retry to stream reconnect method
f4cd068
@tsmith023
Remove timeout and retries from new grpc methods
8912f86
@tsmith023
Only delete key if present in dict
450c010
@tsmith023
Close before re-connecting, reset rec num objs on shutdown
1ab42cd
@tsmith023
Update to use latest protos and behaviour
d214507
@tsmith023
Improve logging using .automatic()
e417701
@tsmith023
Update CI image to latest server build
3cfb194
@tsmith023 tsmith023 changed the base branch from main to dev/1.33 August 26, 2025 12:44
orca-security-eu[bot]
orca-security-eu bot reviewed Aug 26, 2025
View reviewed changes
Copy link

@orca-security-eu orca-security-eu bot left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca

@tsmith023 tsmith023 mentioned this pull request Aug 27, 2025
Merged
4 tasks
@tsmith023 tsmith023 changed the base branch from dev/1.33 to main October 10, 2025 11:41
orca-security-eu[bot]
orca-security-eu bot reviewed Oct 10, 2025
View reviewed changes
Copy link

@orca-security-eu orca-security-eu bot left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Infrastructure as Code high 0   medium 0   low 0   info 0 View in Orca
Passed Passed SAST high 0   medium 0   low 0   info 1 View in Orca
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Vulnerabilities high 0   medium 0   low 0   info 0 View in Orca
🛡️ The following SAST misconfigurations have been detected
NAME FILE
info Weak random number generation may compromise cryptographic security ...ing/test_shutdown.py View in code

tsmith023 and others added 25 commits October 13, 2025 13:28
@tsmith023
Update to match new server impl
ed38170
@tsmith023
Update to use latest server image
8f5f1b1
@tsmith023
Only start threads once to avoid runtime error when handling shutdown
63717f5
@tsmith023
Update CI images
f1d441b
@tsmith023
Hard-code SSB concurrency to 1 for now
92572d4
@dirkkul
Fix collection.batch.automatic
cb8bbab
@tsmith023
Correct logic in _BgThreads.is_alive
a9d8fc2
@tsmith023
Adjust default batch size to align with server default and avoid over…
60cead8 
…loading server too fast
@tsmith023
Update CI images and version checks in tests
3c0842e
@tsmith023
Update to use latest server behaviour around backoffs and uuid/err re…
f60ba97 
…sults
@tsmith023
Lock once when reading batch results from stream
b027030
@tsmith023
Interpret context canceled as ungraceful shutdown to be restarted by …
76b758d 
…client
@tsmith023
Use backoff message to adjust batch size
abb34c6
@tsmith023
Start batching with smallest allowed server value
f400c3f
@tsmith023
Add extra log in batch send
88ee682
@tsmith023
Reintroduce timeout when getting from queue
0acafb0
@tsmith023
Add log to empty queue
56cdd2b
@tsmith023
Add log to batch recv restart
0a2e402
@tsmith023
Remove timeout when getting from internal queue
9510b59
@tsmith023
Only update batch size if value has changed
9523e4d
@tsmith023
Track then log total number of objects pushed by client
13ebff6
@tsmith023
WIP: receive shutdown as message and not rpc error
fa2ef0d
@tsmith023
Move result writing inside message.results case
6acf173
@tsmith023
Add missing proto changes
04766fd
@tsmith023
Update CI image
4612966
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@orca-security-eu orca-security-eu[bot] orca-security-eu[bot] left review comments

@dirkkul dirkkul Awaiting requested review from dirkkul

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@tsmith023 @codecov-commenter @weaviate-git-bot @dirkkul